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In 1999, the Joint Advanced Warfighting Program (JAWP) at the Institute for Defense 
Analyses (IDA) conducted with and for the US Joint Forces Command its first joint ex¬ 
periment. This first experiment had multiple objectives. Its substantive focus was on explor¬ 
ing new joint concepts to detect and attack critical mobile targets, specifically theater 
ballistic missiles. (The description of the experiment design and results is covered in a US 
Joint Forces Command document, The Joint Experiment J9901: Attack Operations Against 
Critical Mobile Targets, September 29, 2000.) This first experiment also estabhshed a base 
of tools and knowledge to explore a broader set of joint concepts necessary to achieving the 
ambitious goals of Joint Vision 2020. 

In addition, a major objective was to learn about designing and conducting effective joint 
experiments. I beheve that this first experiment was successful in ma turing the joint ex¬ 
perimentation process by creating a disciplined environment conducive to both discovery 
(what could be) as well as measurement (what happened). The attached paper offers a menu 
of insights and observations that the JAWP team gleamed from its experiences with J9901 
(which relied heavily on human play in a computer-generated synthetic environment). I 
hope it will be helpful to others charged with the task of designing and conducting future 
experiments. 

I invite your comments and feedback, which should be directed to: 

IDA-JAWP 
ATTN: John Fricas 
1801 North Beauregard Street 
Alexandria, VA 22311-1772 
Telephone: 703-845-6689 
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Preface 


This document was prepared for the Director, Defense Research and Engineering, in 
the Office of the Under Secretary of Defense for Acquisition, Technology, and Lo¬ 
gistics, under the task order Joint Advanced Warfighting Programs (JAWP). It ad¬ 
dresses the task order objective of generating advanced joint operational concepts 
and joint experimentation to assist the Department of Defense in attaining the objec¬ 
tives of Joint Vision 2020. Members of the JAWP contributed to the ideas and re¬ 
view of this document. Other contributors included Dr. Ronald A. Enlow, Dr. John 
E. Morrison, Dr. Dennis F. DeRiggi, and Mr. George E. Lippencott. 

The JAWP was established at the Institute for Defense Analyses (IDA) by the Of¬ 
fice of the Secretary of Defense and the Joint Staff to serve as a catalyst for 
stimulating innovation and breakthrough change. The JAWP Team is composed of 
military personnel on joint assignments from each Service as well as civilian ana¬ 
lysts from IDA. The JAWP is located principally in Alexandria, Virginia, and in¬ 
cludes an office in Norfolk, Virginia, that facilitates coordination with the United 
States Joint Forces Command. 

This document does not necessarily reflect the views of the Institute for Defense 
Analyses or the sponsors of the JAWP. Our intent is to stimulate ideas, discussion, 
and, ultimately, the discovery and innovation that must fuel successful transforma¬ 
tion. 
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1. Introduction 


1.1 Background 

On May 15, 1998, the Secretary of Defense designated the Commander-in-Chief of the US 
Joint Forces Command (formerly US Atlantic Command) as his executive agent for joint 
warfighting experimentation. Joint Forces Command assumed the mission on October 1, 
1998. 

Between October 1998 and August 1999, the Joint Advanced Warfighting Program (JAWP) 
at the Institute for Defense Analyses (IDA) designed, prepared, and conducted the first joint 
experiment for Joint Forces Command under its new charter. The experiment received the 
designation as Joint Experiment J9901: Attack Operations Against Critical Mobile Targets. 
The J9901 experiment had three major objectives: 

► Explore a new concept for attacking mobile targets. The essence of this “Attack Op¬ 
erations” concept was an adaptive command and control (C2) arrangement that em¬ 
ployed a sufficiently dense and diverse suite of sensors, coupled with responsive 
weapons, to enable the anticipation, tracking, and targeting of these elusive targets. 

► Serve as an exemplar to increase the understanding of how to conduct joint experi¬ 
ments, including the use of human-in-the-loop (HITL) simulations, that can produce 
breakthrough changes in future joint force capabilities. 

► Lay a groundwork (tools, people, and knowledge) for diverse future experiments, 
especially joint C2 issues. 

Joint experimentation is a new endeavor—thus, it is not surprising that there are a variety of 
views about what it is and how to do it. Two attributes m particular characterized J9901. It 
featured HITL within a synthetic environment. It also emphasized discovery and learning, 
not only allowing but encouraging adaptation and innovation by the players over the course 
of the experiment. Not all future joint experiments will feature these aspects as prominently. 
Some of the observations relate specifically to these two features; most, however, are more 
general in nature. 




1.2 Purpose of This Document 

This document offers observations and lessons learned about the methodology and proc¬ 
esses used in preparing for and executing the Joint Experiment J9901. Another document, 
The Joint Experiment J9901: Attack Operations Against Critical Mobile Targets, separately 
covers the results of the experiment.^ The J9901 Experiment Design Document, the J9901 
Experiment Management Plan, and the J9901 Trials Repository all contain additional 
documentation, including the design and implementation of the experiment. 

The primary focus of these observations and lessons learned is to assist those involved in 
designing and conducting future joint experiments. Even with this audience in mind, some 
of the observations may appear to fall in the category of “the obvious.” They were included 
because, retrospectively, many were not obvious to the JAWP team at the time they were 
putting J9901 together. We note four principal lessons learned: 

► The operational concept is the heart of the experiment and should remain so 
throughout the entire process. The concept should be described in sufficient detail 
and early in the process to facilitate its evolution and maturation. The centrality of 
the operational concept to the experiment can too easily get lost in the press of mod¬ 
eling, simulation, data collection, and all the other necessary supporting activities. 

► Human behavior is an essential component of warfare, and thus the joint experiment 
must capture this human dimension. Experiments about joint C2 (largely about deci¬ 
sion making) should have the support of HITL simulation tools. (J9901 showed the 
practicality of the HITL simulation tool in joint experimentation.) 

► There is a need for a top-notch simulation team along with the simulation tools. One 
cannot do successful joint experiments with a “pick-up” simulation team. 

► An adaptive, thinking, and creative Red Team is critical. Considerable time and re¬ 
sources are required to make this happen. 

1.3 How to Use This Document 

Chapter 1 pre.sents an overview of the background, tasking given to the JAWP, and high¬ 
lights the four principal lessons learned. 
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Chapter 2 presents a brief overview of the joint experimentation process, discusses the 
phases that make up the process, and describes the JAWP team’s approach to the design and 
conduct of the J9901 experiment. 

Chapter 3 presents a compilation of comments that provide observations and lessons 
learned about joint experimentation in general—and J9901 in particular. These include 
things that went well and things that did not go well. The comments represent the collective 
judgment of the team that conducted J9901. All were intended to help future experimenters, 
including those without extensive mihtary or joint experimentation experience. 

The appendix contains a more detailed description of the J9901 Experiment. References 
and acronyms are provided at the end of this document. 




2. Designing the J9901 Experiment 


This chapter presents a brief overview of the joint experimentation process, discusses the 
phases that make up the process, and describes the JAWP team’s approach to the design and 
conduct of the J9901 experiment. 

2.1 The Joint Experimentation Process 

The experimentation process is depicted in Figure 1. A description of each phase of the 
process follows. 



Figure 1. Experimentation Process 


Concept Development. As discussed in the US Joint Forces Command report on J9901, 
the joint experimentation process used was iterative, beginning with the initial concept de¬ 
velopment.^ 


The experimentation process depicted is intended to be flexible and is not linked to a set series of detailed 
procedures and functions. Consequently, Figure 1 serves primarily as an illustration of the overall process 
and should not be interpreted as a formula for a step-by-step meAodology. It also serves to aid the proc¬ 
ess of discovery in a disciplined manner. 
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Discussions and War Games. Once initially developed, participants evaluated the concept 
in seminars and simple tabletop war games. The goal was to ensure that no “show-stoppers” 
were evident in the concept at the beginning of the process. As a result of the seminars and 
war games, changes were made. 

Constructive Simulation. The next step in the process was a more in-depth look at the 
concept using a constructive simulation that could identify possible strengths and weak¬ 
nesses as well as provide additional insights. Refinements were made to the concept. These 
included setting the initial parameters and numbers for sensors and other systems used in 
the virtual HITL simulation. 

Virtual HITL Simulation. This phase introduced human players into the experiment, 
which allowed for the following: 

► exploration of human ability to manage the concept, 

► investigation of the dynamic interaction between opposing forces (Blue vs. Red), 

► assessment of human contribution to the concept, and 

► identification of weaknesses and/or improvements to the concept. 

Phases Not Involved. The remaining two possible phases in the experiment process could 
involve live field events and actual real-world operations, but J9901 did not continue fur¬ 
ther in the experimentation process because of time, personnel, and funding constraints. 
Additional analysis, constructive simulation, and HITL simulation would then follow these. 

Note: At any point along the process, the concept could have been redesigned or reworked. 
Failure of the concept was an acceptable outcome because even failure provides for discov¬ 
ery and learning. 

2.2 The J9901 Approach 

The J9901 experiment was concept based and used a Red Team whose job was to defeat the 
concept with forecasted adversary technology, tactics, and techniques. The goal was not to 
specifically attempt to either control independent variables or conduct trials of sufficient 
numbers to be able to provide statistically significant data. J9901 was a heuristically guided 



investigation to support discovery and innovation.^ The focus was on learning how to con¬ 
duct attack operations and not on evaluating how many pieces of a particular system one 
might procure for this mission. 

Several factors drove the specific experiment approach and design of J9901: 

► First, it represented the beginning of a new experimentation program; consequently, 
there was a need to investigate ways to do future experiments. 

► Second, the specific problem selected—attack operations against theater ballistic 
missiles (TBMs)—represented an enduring problem. 

► Third, there were several previous efforts dealing with the problem that provided a 
wealth of knowledge and information for experiment development. 

► Finally, the Joint Semiautomated Forces (JSAF) program offered the HITL model¬ 
ing and simulation tools for building an experiment that looked into the future. 

The availability of resources impacted the design and conduct of the experiment— 
principally time, funding, and personnel. The US Joint Forces Command required that the 
JAWP conduct the experiment, from concept development through trial execution within 
Fiscal Year 1999 (October 1, 1998, through September 30, 1999). These resource con¬ 
straints, however, did not prevent the execution of an effective first joint experiment 


^ See D. Robert Worley, Defining Military Experiments, Institute for Defense Analyses, Alexandria, VA, 
February 1999. 



3. J9901 Experiment: Lessons Learned 


This chapter presents a compilation of observations and lessons learned about joint 
experimentation, based on the J9901 experience. The comments represent the col¬ 
lective judgment of the team that conducted J9901, and cover both successes and 
shortfalls in the J9901 experimentation process. 


We have organized this compilation into the following 11 areas that would have to 
be addressed by those designing and conducting future experiments: 


Experiment Schedule 
Concept Development 
Experiment Design 
Discussions and War Games 
Training 

Modeling and Simulation 


Red, White, Blue, and Gold Teams 

Experiment Execution 

Assessment and Analysis 

The Experiment Team 

Funding and Contractual Arrangements 


3.1 Experiment Schedule 

Figure 2 depicts the J9901 experiment schedule. 
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Figure 2. J9901 Experiment Schedule 




In view of the given time constraints, J9901 was an ambitious project—from start to 
finish in less than a single fiscal year. An experiment of this size and scope should 
have been executed over an approximately 18-month period. Numerous tradeoffs 
were made to accommodate the time window. The additional time—six months— 
could have been used to develop, prepare, and mature the concept. It would have 
allowed for additional constructive simulation runs that would have directly bene¬ 
fited and informed the conduct of the actual trials. A longer experiment schedule 
would have permitted a longer break between trials. This would have enhanced each 
subsequent experiment by allowing changes to be made in the simulation to ac¬ 
commodate emerging insights or fix problems encountered. 

In general, time will likely be the most precious resource, and too little of it will be 
available for in-depth exploration of complex concepts. Therefore, the concept devel¬ 
opment and experimental design teams must be explicit in defining an experiment’s 
focus that is compatible with the time available. They must also provide a credible 
context for this focus that establishes hooks to other ongoing and future experi¬ 
ments. In J9901, for example, robust air defenses and the need for extensive Joint 
suppression of enemy air defenses (JSEAD) were assumed away and their effects 
left for subsequent experimentation. 

Experiment milestones must be established early. The schedule must include plans 
for in-progress reviews, holidays, travel, set-up time, VIP visits, etc. The process of 
“discovery” experimentation also demands an adaptable schedule; therefore, flexi¬ 
bility is key throughout the planning process or else experimenters will be forced to 
work at an impractical pace. 

Experiments on significant issues will naturally draw interest from audiences out¬ 
side the experiment and generate requests to visit and observe the experiment while 
it is underway. This requii'es planning to provide both for quality exposure of the 
experiment to the visitors while simultaneously ensuring their presence does not dis¬ 
rupt the experiment. The issue of visitors must be addressed early to ensure that im¬ 
portant audiences have the opportunity to schedule their visits in a fashion that is 
manageable and that the design of the experiment (both layout and process) can ac¬ 
commodate their presence. 

Early identification and coordination of simulation and facility requirements are es¬ 
sential. J9901 was fortunate to be able to utilize (1) an existing simulation (the Syn- 
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thetic Theater of War Advanced Concept Technology Demonstration (ACTD), sub¬ 
sequently renamed JSAF) and (2) a sponsoring facility, the US Joint Forces Com¬ 
mand’s Joint Training, Analysis and Simulation Center, to conduct an activity that 
had not been factored into anyone’s long-range planning. Such good fortune should 
not be anticipated for future experiments. 

In developing an experiment schedule that crosses fiscal-year boundaries, provisions 
are necessary to ensure adequate funding is available across the fiscal-year boundary. 
Provisions can be made to ensure continuity and a smooth transition between fiscal 
years. On the other hand, a lack of planning could cause significant problems, such 
as the inability of participants to travel because of the temporary unavailability of 
TDY funds. 

3.2 Concept Development 

Before the development began, the J9901 Team was assisted by the fact that attack op¬ 
erations were already receiving significant attention within the Defense community, 
with several studies extant or underway. The team putting the J9901 concept together 
used these studies and activities to gain credibility and to leverage the work already 
completed. 

Although J9901 involved research into technology programs at the Defense Ad¬ 
vanced Research Projects Agency (DARPA), the National Laboratories, the Lincoln 
Laboratory at the Massachusetts Institute of Technology, and some commercial 
firms, the team did not cast its net wide enough. If time had been available, further 
attempts could have incorporated Service laboratories and other academic institu¬ 
tions. In future experiments, there is a need for several technical symposia to bring 
together interested parties from academia, government, and commercial sources to 
ensure that the best information is available to concept designers. 

The presence of a high-level advocate or stakeholder during concept development and 
implementation would have helped the Concept Development Team over the inevitable 
obstacles in the way of achieving a full and comprehensive understanding of a mission 
area. The presence of the stakeholder is also necessary to ensure that outcomes of 
the experiment will find their way into the transformation process. 

Concepts should not be set in concrete at any point during the experiment In J9901, a 
more detailed concept evolved over the course of the experiment, based on the dis- 




covery and learning that took place from trial to trial. This would not have occurred 
had a more rigid approach been taken. 

Although concept development, experiment design, and simulation designs are logi¬ 
cally sequential and separate, in fact, they are functionally interdependent, overlap¬ 
ping, and iterative. Nevertheless, concept development should be the driver. Concept 
development must occur first to provide direction. 

During the experiment development phase of J9901, the Concept Development 
Team focused most of its efforts on sensors, weapons, automatic target recognition 
(ATR), tracking, and fusion. However, even though innovative joint C2 was an es¬ 
sential element of the concept, serious attention to C2 issues was delayed until later 
in the process. Consequently, requisite efforts were not devoted to the development 
of C2 displays and decision aids, particularly the capabilities resident in future 
workstations and the organization of the Critical Mobile Target Cell (CMTC). We 
were fortunate that the JSAF simulation team did cover this area on its own initia¬ 
tive. Command and control will likely be central to future joint experiments and 
should be considered up front The Concept Development Team needs to devote time 
and energy to the areas of displays and decision aids to ensure that reali.stic future 
capabilities, including human-computer interfaces (HCIs), are built into the C2 sys¬ 
tem from the beginning. 

The Red Team cannot limit itself to merely asking the intelligence community for 
the requisite estimates. The Red Team must work with technologists, academia, the 
National Laboratories, and even commercial firms to identify these future capabilities. 
The Red Team should cast its net widely, identify sources of expertise, and keep these 
sources available during the experiment The Red Team must contain its share of the 
best people available. 

Close interaction with the Services can enhance joint experimentation. Service in¬ 
teraction was not extensive in J9901 and came primarily in the form of concerns for 
why particular Service programs were not represented. Consideration should be 
given to greater Service involvement in future experiments, with the most desirable 
course being their involvement throughout the process from beginning to end. 



3.3 Experiment Design 

The experiment design must be flexible to deal with surprises, changes in the con¬ 
cept, radical shifts in technology, as well as innovative Blue or Red tactics and doc¬ 
trine. 

In J9901, assumptions made during the design phase had an effect on the whole proc¬ 
ess of experimentation. Examples include assumptions about probability of detec¬ 
tion, automated tracking and ATR performance, sensor false alarm rates, and 
weapon performance. Some time and effort need to be dedicated to understanding 
these assumptions and their impacts on the experiment. For example, the assump¬ 
tion of a low false alarm rate would significantly impact sensor performance. On the 
other hand, setting the false alarm rate too high would significantly alter the results. 
The constructive simulation phase can be invaluable in identifying and understanding 
the potential impact of assumptions such as the sensor false alarm rate, sensor area 
coverage, and ATR performance. 

In J9901, each trial (except for Trials F and H) covered a 24-hour period. Because of 
resource constraints, it was a challenge to set up runs of 24-hour duration. Neverthe¬ 
less, it was understood that this timeframe was not long enough to experiment fully 
with the concept. For example, a 48- or even 72-hour period would have allowed for 
a greater exploration of the impact of attacking TBM infrastructure and logistics. 
Longer trial periods would also have allowed for better understanding of the impact 
and use of long-dwell sensors. 

The basic concept and design for the experiment need to be developed and docu¬ 
mented in as much detail as possible—and as early as possible—^in the Experiment De¬ 
sign Document. Subsequently, the concept and experiment design can adjust as 
planning proceeds, and can be changed as late as the beginning of the trials. 

Advocates for the key components of the concept must be identified early to ensure 
that the capabilities of their components are accurately simulated and utilized during 
the experiment. Examples of these key components include sensors, C4I systems'*, 
and weapons. Every effort should be made to keep these advocates engaged and in¬ 
volved throughout the experimental process. 
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Command, control, communications, computers, and intelligence. 




The experiment’s data collection and analysis plan (DCAP) is an integral part of the 
design. It cannot be created retroactively. 

In J9901, analysts largely created the DCAP because they were more familiar with 
the analytical process. The team putting the DCAP together should have also included 
members with experience in military operations, intelligence, missile threats, sensors, 
and weapons. This would have led to a more effective DCAP and would have al¬ 
lowed for other members of the team to become familiar with and understand the 
DCAP development process and its impact on the experiment. 

The experiment design must include multiple trials to fully explore the concept. 
However, determining the number of trials is difficult. In J9901, the decision was 
for two rehearsal trials and seven record trials (the first two of which were less use¬ 
ful because of problems with data from the After Action Review System (AARS)). 
More use of the constructive simulation to explore the concept in more detail prior to 
the HITL trials would have provided a better basis for selecting the number of HITL 
trials. 

Selecting the parameters to vary in the experimental trial will present a challenge to 
the experiment designers. The nature of most experiments—their complexity and 
two-sided human involvement—will make it difficult to isolate single variable cau¬ 
sality. The use of constructive simulations (even more extensively than we were able 
to do in J9901) that are closely linked to the HITL trials can help in shedding light 
on these relationships. Another complication arose during J9901 and will likely 
again in future experiments. During the course of the experiment, plans will change 
and parameters and variables will need to be adjusted—sometimes even during an 
individual trial. The reasons range from problems with the simulation and models to 
reacting to what is being learned during the experiment. We strongly suggest that a 
process be established before the trials start to deal with these “unexpected” 
changes, and to consider the alternatives and consequences. 

More wargaming of the operations concept, using outsiders who have operational ex¬ 
pertise, is useful for identifying and correcting problem areas. This did not occur be¬ 
cause of time constraints. Problem identification and fixes had to be made during 
the trials. Additional wargaming could have prevented this. For example, there were 
no provisions for the timely reporting of battle damage assessment (BDA) to the 
players. This became a central issue for the players after the fust two trials. BDA 



reporting was added and required the dedication of an individual to provide “man¬ 
ual” feedback to the CMTC during the remaining trials. 

Security planning must play a major role from the beginning in the design and con¬ 
duct of an experiment Even unclassified experiments need to prepare security 
guidelines. The absence of proper security policies and procedures can delay or dis¬ 
rupt the experiment. Classification policies and procedures, including requirements 
for classified communications support and security clear procedures for visitors, 
need to be drawn up at the beginning of the experiment design process. 

There is a need for long lead-times to get approval for foreign officer participation in 
any Department of Defense (DoD)- or Service-sponsored military experiment The 
clearance and approval process must begin immediately once it has been determined 
that allied military or personnel from other foreign government agencies will par¬ 
ticipate in the experiment. 

3.4 Discussions and War Games 

Early detailed discussion is an essential step for defining the concept, identifying the 
key issues, selecting the appropriate experimentation tools, and identifying expertise 
and resources. In addition to much internal discussion, there were two organized 
discussion sessions involving outside personnel. The first, hosted by the Joint Thea¬ 
ter Missile Defense - Attack Operations Joint Test and Evaluation, occurred at Kirt- 
land Air Force Base, New Mexico; and the second, which included representatives 
of the National Laboratories, at IDA Although these sessions were very useful, 
there were only two, and these were limited in scope. Additional sessions with a 
more inclusive group probably would have been useful in “fleshing out” the overall 
experiment. 

Because of hmitations on time, there was no formal wargaming conducted in prepa¬ 
ration for J9901. Several war games involving operational players would have been 
useful. However, if the right players are not selected, i.e., those who are capable of 
thinking “outside the box,” there is a danger that “breakthrough” concepts will be 
overlooked in favor of doing things in line with current practices. 

Several “Transparent War Games” prepared the Red Force for the experiment. These 
war games consisted of iterative open sessions by the Red and Blue Design Teams 
to understand each other’s concepts and devise possible countermeasures. The 





Transparent War Games were helpful in fleshing out each side’s tactics and ensuring 
that the simulations could model them. 

3.5 Training 

There is a need for a training plan tailored to the players. The experience level of the 
players must be considered in the length of the training. In J9901, the time allocated 
for training—two weeks—was not sufficient. Based on player observations during 
the early trials, more time would have been profitable for hands-on training. 

In J9901, software developers were needed for effective training. Their contribution 
was significant and provided a critical enhancement to the quality of training. This 
benefit, however, must be balanced with the opportunity lost for their primary duty 
of software development. Consequently, care must be taken to ensure there are suffi¬ 
cient people dedicated to the training role and software development to prevent inter¬ 
ference with either the accomplishment of the training mission or the timely delivery 
of software. 

Discovery experiments, by their very nature, are dynamic and require flexibility to 
maximize the learning experience. One should expect to make adjustments and re¬ 
finements during the experiment. Additional training most hkely will be required as 
a result of the adjustments and refinements. Therefore, training must be planned as a 
continuous process. 

Assessors running the assessment system, members of the Red and Blue Teams, 
simulation operators, experiment controllers, and individuals conducting VIP brief¬ 
ings must receive training to understand the experiment and supporting systems. 
Training time to accomplish this must be integral to the training schedule. 

The collection, development, and archiving of quality training materials and products 
are significant efforts in their own right. They must start early and can end up driving 
production of substantive aspects of the experiment. The limited time available to 
develop J9901 made this a hectic and high-risk operation. In contrast, the training 
for Trail H in April 2000 benefited significantly from the experience and materials 
gained in the conduct of earlier trials. This reinforces the importance of archiving 
experiment materials from experiment to experiment that may have utility more 
than once. 



3.6 Modeling and Simulation 

Human behavior is an essential component of warfare and, consequently, of joint 
experimentation. Entity-level HITL simulation, as demonstrated in J9901, permitted 
the exploration of the human dimension of the concept. J9901 further underscored 
future need for high fidelity HITL simulations as an essential component of joint ex¬ 
perimentation. 

Based on the J9901 experience, simulations for joint experimentation should: 

► have a modular structure, enabling easy modification of the scenario, con¬ 
cept of operations, simulated forces, systems, technologies, and data; 

► be iterative and/or adaptable, allowing shifts from constructive to virtual to 
war games to seminars to mixed environments as new questions emerge; 

► be quick turning, keeping up with the constant shift of Red and Blue tactics, 
and emphasizing the approximate right answer rather than the precise, in¬ 
complete (and often wrong) answer; 

► be High-Level Architecture (HLA) compliant, capable of federating with joint 
forces and/or capabilities that are Hve, virtual, and constructive models; and 

► undergo verification, validation, and accreditation for future innovative con¬ 
cepts based on operational judgment and reasonableness. 

Validation and verification (V&V) of entity and model performance are one of the 
most demanding and time-consuming aspects of using modeling and simulation in ex¬ 
perimentation. It should start early with the identification of the appropriate subject 
matter experts who wiU eventually validate the effort and seek continuous involve¬ 
ment throughout the design and development of the experiment. V&V of future 
concepts, technologies, and tactics must necessarily rely heavily on operational 
judgment because, in most cases, the systems do not exist and/or the concepts have 
not been tested. V&V must take place at least six weeks in advance of the start of 
trials so that corrective action may be taken without delaying the start of the ex¬ 
periment. It should have been done in more detail and earher in the process. 

A Modeling and Simulation Annex to the Experiment Design Document is essential. 

This annex should include a Model Requirements Document that lays out in great 
detail the requirements for the simulation. The Model Requirements Document must 




contain a Joint Entity List that describes each entity in the simulation and the behav¬ 
iors and characteristics associated with that entity. This documentation is necessary 
not only for V&V but also for the post-experiment analysis. 

In J9901, there was a significant requirement for performance data to enable the simu¬ 
lation developers to model the software objects representing such components as new 
weapons systems and sensors. To keep simulation development on schedule, this per¬ 
formance data must be available to the simulation developers at least 90 days prior 
to integration testing. The performance characteristics must have documentation to 
support the simulation V&V. In J9901, this was handled by the development of a 
Joint Entity List that contained the detailed performance characteristics of each of 
the systems modeled in the experiment and details on how these characteristics were 
represented in the software. 

3.6.1 Virtual HITL Simulation 

For simulated experiments, a quality simulation team is absolutely essential. This 
team must be capable of operating the simulation and making fixes and improve¬ 
ments as needed in a short period of time. Ideally, the simulation team should also 
include the developers who built the version of the simulation used for the experi¬ 
ment. In J9901, the JSAF Team had been together for at lea.st four years and had 
participated hi two major simulation exercises. The developers who were part of the 
team were thus able to make changes in a timely fashion and anticipate require¬ 
ments in advance. You cannot do successful joint experiments with a pick-up simula¬ 
tion team. 

JSAF was the best available simulation to meet the J9901 HITL simulation objec¬ 
tives. It was, however, difficult to change the scenario or templates given the time 
constraints of the experiment. Because discovery experiments invariably lead to 
changes in the simulation, either the model needs to be flexible or the supporting pro¬ 
grammers need to be flexible. 

Simulations used in joint experimentation must have the support of extensive docu¬ 
mentation. An alternative is to have the simulation developer available during the 
experiments to make changes and to troubleshoot problems. In J9901, the Simula¬ 
tion of Location and Attack of Mobile Enemy Missiles (SLAMEM) developers were 




available to troubleshoot problems quickly—their presence saved the experiment on 
several occasions. 

In J9901, two weeks were scheduled between trials to make changes to the model 
and fix problems encountered in the trial. These changes often required a “full court 
press” by the simulation team so that the changes and fixes would be ready for the 
next trial. This did not allow enough time for testing the fixes. Three weeks between 
trials would have been better. 

In J9901, because of time constraints, there was insufficient time for testing the 
simulation. Although sub-system testing had gone on for several months, r unnin g 
the full simulation under load only took place in the two weeks preceding the start 
of training. The AARS was not run under full load prior to the first trial and, as a re¬ 
sult, problems with the AARS corrupted the data from the first two trials. The prob¬ 
lems encountered were not the fault of the JSAF Team or the AARS Team. Both of 
these teams worked very hard to get the simulation ready in a relatively short time, 
given all of the changes required to support the J9901 concept. 

By its nature, joint experimentation will require significant changes to any existing 
simulation since new concepts, new systems, new technologies, new doctrines, new or¬ 
ganizations, and new tactics are under investigation in every experiment—sometimes 
in every trial. Experiment designers must allow for this. An optimum solution would 
be to have the simulation completely ready two months prior to the start of training 
to provide time for full-up load testing, V&V, and the inevitable fixes and re-testing. 


3.6.2 Constructive Simulation 

The constructive simulation must be available for concept development work early on. 

In J9901, there was only sufficient time prior to the HITL trials to use SLAMEM to 
investigate the sensor and weapons parameters for the HITL simulation, as well as 
look at the numbers of satellite, unmanned aerial vehicle, and attacks systems re¬ 
quired. This simulation, however, did not allow for dynamic interactive play be¬ 
tween Red and Blue Forces. Changing Blue and Red systems and responses could 
not be done on the fly. 

In J9901, federating the JSAF simulation and the SLAMEM constructive simulation 
worked well. 




3.7 Red, White, Blue, and Gold Teams 


The concept of an adaptive opposing Red Force is vital to discovery experimentation. 

Consequently, the Red Force must retain a significant degree of freedom to operate 
and make independent decisions. In J9901, however, the Red Force did not possess 
a total free rein on its actions. Rather, Red actions were closely coordinated between 
the Red Team Leader and the White Team Leader to meet experiment objectives and 
to avoid unnecessary disruptions. 

The Red Team had two primary functions: (1) to help develop “counter concepts” 
and define future adversary capabilities, and (2) to serve as the reactive opposing 
force during the trials. The work done by the Red Team occurred without benefit of 
allied or foreign inputs. Thus, potential adversary cultural and other biases were not 
reflected. Getting inputs from allies or members of other foreign militaries, as well as 
getting foreign role players on the Red Team, would be very useful in future experi¬ 
ments. 

J9901 focused on future capabilities circa 2015. It is vital that both Blue and Red 
Force capabilities reflect the same time frame. (There is a tendency to give Blue 
Force a future capability paired against current day Red capability.) 

Experiment control mechanisms need to be established early. These must include con¬ 
trols over Blue and Red Teams to maintain the experiment within agreed bounds. 
Scenario-tracking procedures also need to be established to meet the experiment’s 
objectives. 

In J9901, the White Team was responsible for the overall control of the experiment. 
The White Team formed late in the concept development phase. Consequently, 
many of the policies and procedures were formulated late. The White Team should 
be formed at the beginning of concept development. White Team activities include 
developing the overall control plan and methods for injecting events and altering the 
scenario, if necessary, to meet experiment objectives. 

To conserve personnel resources, members of the simulation team also doubled as 
members of the White Team. The downside to this arrangement was that then duties 
sometimes conflicted with then role as members of the White Team. On the plus 
side, then expertise in simulation was essential to the successful operation of the 




White Team function. Future experiments should avoid the “dual hatting” of the 
White Team members. 

A Gold Team should be formed early to observe and evaluate the experiment from an 
external and independent perspective. It should seek insights on both the substance 
and design to identify flaws or weaknesses to which those closer to the execution 
may be oblivious. This effort should start early and be continuous throughout the 
experiment. The goal of the team is to provide constructive criticism to improve 
both (1) the design and conduct of the immediate experiment and (2) the design and 
execution of future experiments. 

3.8 Experiment Execution 

In J9901, the same group of players was retained through the first nine trials of the 
experiment. Although the experiment designers recognized that there would be sig¬ 
nificant learning effects over the course of the experiment, they felt that the in¬ 
creased competence of the players would result in their being able to provide 
valuable insights into the goals of the experiment. This was borne out by the ex¬ 
periment. We were also fortunate in having a number of capable, experienced per¬ 
sonnel as players—which underscores the fact that it is essential to recruit the right 
players. 

The plan for experiment execution must recognize that the experiment day is longer 
than just the simulation time. J9901 was designed with the belief that six hours was 
the maximum amount of time players could effectively sit in front of their terminals. 
Before and after each day’s simulation run, time was needed for a mo rnin g situation 
briefing, bringing up the simulation, training (as required), daily “hot washes” 
and/or after actions reviews, and planning for next day’s activities. 

During the development phase of J9901, the major elements of the experiment team 
and the simulation team were geographically separated from the US Joint Forces 
Command J-9 elements in Suffolk, Virginia. However, coordination between and 
among these distributed elements was imperative and required on a regular basis. 
The use of a weekly video teleconference between the team members in the Washing¬ 
ton area and those in the Suffolk area proved productive. 



The scenario for the experiment required specific key inputs before the begin¬ 
ning of each trial. For J9901, these elements for Blue and Red included the follow¬ 
ing: 

► a force list (including organization, unit structure, and strength); 

► a force laydown (examples include C2, unit/platform posture, logistics, posi¬ 
tion); 

► rules of engagement; 

► intelligence inputs and intelligence preparation of the battlespace (IPB) 
products; 

► commander’s guidance (including priorities, collateral damage limits, and 
employment criteria); 

► mission objectives, constraints, and desired end state; 

► operations order; and 

► unit and/or force standing operating procedures (doctrine; tactics, tech¬ 
niques, and procedures (TTPs); reporting) 

3.9 Assessment and Analysis 

In J9901, the players received great latitude to organize themselves in the most effi¬ 
cient manner and to develop the most efficient TTPs to prosecute targets. As a re¬ 
sult, continuous HITL assessment of the players was essential for understanding the 
changes which the players made and, more importantly, to understand the reasons for 
the changes. HITL observers should remain with the players full time to capture 
theh thought processes during the experiment. 

For J9901, the plan was for ongoing as well as retrospective assessment and analy¬ 
sis. The tools for the ongoing analysis—including those used by the HITL asses¬ 
sors—were developed late and matured as the experiment progressed. Problems 
with the AARS during Trials A and B caused an invalidation of those trials because 
the data collected was corrupted. Consequently, the primary retrospective analysis 
tool, which was the AARS, came on line late. The AARS improved as the trials pro¬ 
gressed. To avoid these problems, the assessment and analysis processes, procedures. 



tools development, and testing should occur as early as possible before the actual ex¬ 
perimental trials. 

HITL assessors need to develop a plan and prepare assessment and/or evaluation tools 
well in advance of the start of the experiment trials. In an HITL experiment, these 
tools must include a vehicle for assessing the decision processes used by the players. 
The tools can then be refined as the experiment progresses. Automated tools, and 
automation in general, can greatly assist in this process. 

Very often, as a result of the experiment, the analysts will make discoveries that will 
require the development of new Measures of Effectiveness/Measures of Perform¬ 
ance (MOEs/MOPs) beyond those developed during the experiment design. Addi¬ 
tional flexibility is needed in the AARS to permit analysts to query the database to 
analyze additional MOEs/MOPs beyond those described in the original experiment 
design. 

A significant amount of data was collected during J9901. The amount of time neces¬ 
sary to digest and evaluate this data was underestimated. In addition, most of the 
team worked on another experiment for approximately two months during the 
analysis and report writing phase. Consequently, writing the final report took longer 
than anticipated. 

The Red Cell, as well as the Blue Cell, requires an assessment and data collection plan 
to understand the changes that occurred in Red TTPs as a result of the Blue efforts. 
The accuracy of the Red data is also key to understanding the Blue data. 

Production of the final report can be enhanced and accelerated if consideration of its 
requirements is a part of the early experiment design and the report is incrementally 
drafted during the experiment Descriptions of experiment activity can be completed 
prior to the end of active experimentation and can be best formulated while events 
are fresh in the writer’s mind, leaving one less item to deal with during the post¬ 
experiment period when attention should be focused on data analysis and findings 
formulation. 

The writing of the final report cannot be left solely to the analysis team. All members 
of the team should participate in the analysis and the development of the final re¬ 
port. 




3.10 The Experiment Team 


For J9901, the JAWP Attack Operations Team acted as the top-level integrator of the 
concept, the simulation, and the experiment. The team did not have enough people 
or expertise to do all of the detailed work needed and relied on the JSAF Team to 
bring the pieces of the simulation together and to operate it. Future experiments will 
also need a more robust “integration” team that includes experts that can bring to¬ 
gether, operate, and manipulate disparate simulations. 

3.11 Funding and Contractual Arrangements 

Many joint experiments will last longer than 12 months. Consequently, there is a 
need for a multi-year funding profile to ensure that the experiment is not delayed by 
lack of funds at the end or beginning of a fiscal year. Funding support that extends 
beyond a single fiscal year, similar to that of DoD’s ACTD program, should be con¬ 
sidered for joint experimentation. 

In addition to a multi-year funding profile, every effort should be made to avoid 
conducting experiments that end at the beginning of a fiscal year. The programming 
and budgeting process can have an adverse impact on the experiment. TDY funding, 
end-of-year funding drills, and having to wait for the allocation of funds for a new 
year can cause problems. 

Because of the short timelines involved in working problems that develop, a flexible 
contractual vehicle to support the experiment is necessary. In J9901, the contracting 
capabilities provided by DARPA’s JSAF program office provided a flexibility. 
However, if DARPA is not a participant, obviously its contracting vehicles will not 
be available. Therefore, a flexible contracting vehicle to support joint experimenta¬ 
tion should be developed. 

Preparation of documentation and reports for J9901 took more resources, time, and 
personnel than anticipated. Realistic funding and other resources need to be allo¬ 
cated early to this process. 

3.12 Concluding Observations 

The methodologies and tools used in designing and executing J9901 proved useful. 
They can be used in future joint experiments comparable to J9901. 




24 




The tools and methodologies used need to be evolved and developed further to support 
the investigation into concepts and problems larger in scope and to support the proc¬ 
ess of continuous joint experimentation. 




Appendix: 

The J9901 Experiment Process 


J9901 generally followed the process outlined in Figure 3/ J9901 did not, however, 
include live field events or real-world operations. Flexibility allowed movement 
within the process. In addition, the constructive simulation phase and the virtual 
HITL simulation phase operated almost in parallel rather than in sequence. This re¬ 
sulted from delays in modifying the constructive simulation, which did not permit 
the fuU use of the constructive simulation in concept development. The constructive 
simulation was, however, used effectively during the concept development phase in 
identifying sensor requirements and performance parameters. 



Figure 3. Experiment Process (Reiteration) 


The J9901 concept development began with an attack operations workshop held in 
Albuquerque, New Mexico, in late June 1998. The concept further matured between 
July and September 1998. During this period the team took advantage of work done 
previously by the Joint Theater Air and Missile Defense Organization, DARPA, the 


^ We provide Figure 1 here again for the convenience of the reader. 








Joint Theater Missile Defense - Attack Operations Joint Test and Evaluation, and 
the National Laboratories. The experiment concept was briefed to the Deputy 
Command-in-Chief, US Joint Forces Command, in September 1998. 

Data gathering, further concept development, and initial work on the experiment 
design and the simulation design occurred between October and December 1998. In 
addition, a Red Team leader was identified, and work on the opposing Red Force 
began in November 1998.^ The first detailed simulation design meeting that brought 
together the planners and the JSAF modeling and simulation team occurred in Janu¬ 
ary 1999. The first meeting with the designers of the constructive simulation also 
occurred in January 1999. Because of modifications to the constructive simulation, 
these runs did not begin at IDA until April 1999. 

The constructive simulation runs used an existing simulation, SLAMEM, developed 
by TOYON Research Corporation. SLAMEM served to explore sensor performance 
parameters, sensor architectures, and concepts for sensor operations for the experi¬ 
ment.^ 

J9901 HITL experimentation began with two rehearsals, conducted between May 3- 
7, 1999, and May 17-21, 1999. These were followed by a series of seven experi¬ 
ment trials that were conducted between June 7, 1999, and August 27, 1999. An 
eighth trial (Trial H) occurred in March 2000 using a different group of Blue play¬ 
ers. The Trial H players consisted largely of active duty Air Force personnel with 
experience in Bosnia and Kosovo Combined Air Operations Center operations. 

More than 1,000 additional constructive simulation runs using SLAMEM occurred 
between September and December 1999 to further investigate relationships uncov¬ 
ered during the post-HlTL experiment trial analysis effort. Analysis and assessment 
of experiment results began with the first trial in June 1999 and continued through 
April 2000. 


Additional infonnation on the Red Team is contained in the US Joint Forces Command docu¬ 
ment, The Joint Experiment J990I: Attack Operations Against Critical Mobile Targets, and in a 
separately published report on the Red Team, Red Teaming: A Means for Transformation, IDA 
Paper P-3580, forthcoming, January 2001. 

’ A more detailed JAWP report, A Constructive Attack Operations Model: Test Results and Analy¬ 
ses, on the use of SLAMEM and the results will be published separately. 



The experimentation process and design are central to the overall joint experimenta¬ 
tion program. Using the process described, J9901 was the US Joint Forces Com¬ 
mand’s first experiment. J9901 represents one approach and can serve as a model 
for joint experimentation for the following reasons: 

► It served to mature the joint experimentation process by creating a disci¬ 
plined environment conducive to both discovery (what could be) as well as 
measurement (what happened). 

► It provided substantive insights into the concept under investigation; how to 
improve attack operations against missile and other critical mobile targets. 

► J9901 also established a base (tools and knowledge) for exploring a broader 
set of command and control and other joint capabilities fundamental to 
achieving Joint Vision 2010. 
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Acronyms 


AARS 

After Action Review System 

ACTD 

Advanced Concept Technology Demonstration 

ATR 

automatic target recognition 

BDA 

battle damage assessment 

C2 

command and control 

C4I 

command, control, communications, computers, and intelligence 

CMTC 

Critical Mobile Target CeU 

DARPA 

Defense Advanced Research Projects Agency 

DCAP 

data collection and analysis plan 

DoD 

Department of Defense 

HCI 

human-computer interface 

HITL 

human-in-the-loop 

IDA 

Institute for Defense Analyses 

JAWP 

Joint Advanced Warfighting Program 

JSAF 

Joint Semiautomated Forces 

JSEAD 

joint suppression of enemy air defenses 

SLAMEM 

Simulation of Location and Attack of Mobile Enemy Missiles 

TBM 

theater ballistic missile 

TDY 

temporary duty 

TTPs 

tactics, techniques, and procedures 

V&V 

validation and verification 


33 




REPORT DOCUMENTATION PAGE 

Form Approved 

0MB No. 0704-0188 

Public reporling burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, 
and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to Washington 
Headquarters Services, Directorate for Information Operations and Reports, 1215 Jefferson Davis Highway. Suite 1204, Arlington, VA 22202-4302, and to the Office of Management and Budget, Paperwork Reduction 
Project (0704-0188), Washington, DC 20503. 

1. AGENCY USE ONLY (Leave blank) 2. REPORT DATE 

October 2000 

3. ] 

] 

REPORT TYPE AND DATES COVERED 

Final 

4. TITLE AND SUBTITLE 

Lessons Learned from the First Joint Experiment (J990I) 

5. FUNDING NO.S 

DASW01-98-C-0067 

AI-8-I627 

6. AUTHOR(S) 

Larry D. Budge, John Fricas 


7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 

Institute for Defense Analyses 

1801 N. Beauregard Street 

Alexandria, VA 22311 -1772 

8. PERFORMING ORGANIZATION 
REPORT NO. 

IDA Document No. D-2496 

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 

Deputy Under Secretary of Defense for Advanced Systems and Concepts 
Director, Defense Research and Engineering 

Office of the Under Secretary of Defense for Acquisition, Technology, 
and Logistics 

3700 Defense Pentagon 

Washington, DC 20301-3700 

10. SPONSORING/MONITORING 
AGENCY REPORT NO. 

11. SUPPLEMENTARY NOTES 

12a. DISTRIBUTION/AVAILABILITY STATEMENT 

Approved for public release, unlimited distribution: Novembei 
2000. 

28, 

12b. DISTRIBUTION CODE 

2A 


13. ABSTRACT (Maximum 200 words) 

In 1998, the JAWP conducted a joint experiment—from concept development through trial execution—for the US Joint 
Forces Command. This document discusses the joint experimentation process and its phases, including the JAWP Team’s 
approach to the design and conduct of the J9901 experiment. Observations and “lessons learned” are provided about 
J9901 and joint experimentation in general, as well as for use in future joint experiments. The JAWP Team noted four 
principal lessons: (1) The operational concept is the heart of the experiment and should remain so throughout the entire 
process. The concept should be described in sufficient detail early in the process to facilitate its evolution and maturation. 
(2) Human behavior is an essential component of warfare; thus the joint experiment must capture this human dimension. 
Experiments about joint command and control (largely about decision making) should have the support of human-in-the- 
loop simulation tools. (3) There is a need for a top-notch simulation team along with the simulation tools. One cannot do 
successful joint experiments with a “pick-up” simulation team. (4) An adaptive, thinking, and creative Red Team is criti¬ 
cal. Considerable time and resources are required to make this happen. 

14. SUBJECT TERMS 15. NO. OF PAGES 

Joint experimentation, attack operations, J9901, simulation, human-in-the-Ioop simulation, 50 _ 

concept design and development, US Joint Forces Command, critical mobile targets. Red le. PRICE CODE 
Team. 


17. SECURITY CLASSIFICATION 18. SECURITY CLASSIFICATION 19. SECURITY CLASSIFICATION 20. LIMITATION OF 
OF REPORT OF THIS PAGE OF ABSTRACT ABSTRACT 


Unclassified 


Unclassified 


Unclassified 


NSN 7540-0 l-280-55(X) 
Prescribed by ANSI Sid. Z39-18 
298-102 


Standard Form 298 (Rev. 2-89) 














